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ABSTEiCTi ' , * , 

This study-assessed the effect of factors outside the 
instructor •s Control on student ratings of instruction* Seven special 
iteas along with the standard s^tudent Eatings fpras were adninisteired 
to 28 classes 'in the Anthropology Departaent.' Eesults indicated that 
the^ impact of outside factorSr as pe^rceived by students^ was not 
correlated with global evaluative ratings* (Author^^C) 
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The purpose of this study was to assess the effa6t of factors 
outnide the instructor/s cdntrol on student ratings of his/her 
instruction. Seven special items^along with the staodard Student 
Ratings forns^were administered to 28 classes in the Anthropology 
Department. -Rftsults indicated that the inqpact of outside rectors, 
as percelvejl by students > was not correlated with global evaluative 
ratings. ^ " . 
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The Effect of Factprs Outelde the Instructot's 
Control on Student Ratings of Instruction 

Gerald M. Gillaore and Richard, W. Naccarato 
INTRODUCTION 



During Spring Quarter, 1974, the Departnent of Anthropology and tha 
Educational Aaaeasnent Center teamed to do research on the effect of ^everal 
variables on atudent/ratlngs of instruction (Gillmpre and Aaoss, Note 1). 
Students In twenty-two classes responded to thlree special items along with 
the standard Student Ratings form. These special items and the response . 
positions are found in Table 1. 

Results of this study showed that classes with higher average rated 
attendance received significantly. higher ratings on laost of the standard 
evaluatiye student ratings iteas. The percentage of assignments cbnq^leted 
waft^not slgnlflcantly\pelated to the evaluative items. Ii^ortantly, the 
third special item exhibited moderately hl^ correlations. Gillaore and 
kaow described this result as follows: 

• 

The direction of the relationship is as expected. Students 
who view factors oqt of the Instructor's control to have a posi- 
tive effect » also fate the cli^s more favora^y* Students who 
are unfavorably liqiressed by factors outside of instructor's^ 
control tand to rate the class less favorably. , Although the 
direction of the relationship is as' expected, the magnitude is 
surprifingly high and potentially ln^iortante If* there 'are factors 
which affect an Instructor's rating which are out of his/her con- 
trol, this is very Important In assessing the meaning of student 
rating results. Furthermore,' if, as the^4ata suggest, the 
students can assess the liq>act of these factors, then it is*- 
fairly sinqple to adjust results appropriately (p. 2). - 

Glllnbre and Amoss also pointed out an alternative explanation for the 
magnitude and direction of the correlations: 



Table 1 % • 
The Tbr^e Special Iteaa of Spring Quarter, 1974, Study 



1. Percentage of da^s neetinga 

^of thia epurae you attended . ^ 
"(Jttrinjg; thia quarter (to the 
lieareat. appros^jaate percent) 90-100 



80-89 



70-79 



60-69 



50 or leaa 



2» Percentage of ^aaaigned 

reading con^leted (to the 
; neareat approxioate percent) 



90-100 



80-89 



70-79 



60-69 



50 or less 



3« Hov havia factors i^cb are 
out of the instructor's con- 
trol ^Uafluenced your evalua- 
tion of thia course (auch aa 
time elaaa aeeta, class room 
location^ class aize, per- 
aonal characteriatics of the 
inatructor, reaaon for 
taking thia course^ etc.)? 



Strong ^ 

Negativje 

tofluence 



Weak 

Negative 

Influence 



No 

Influence 



Weak 

Positive 

Influence 



Strong 

Poaitive 

Influence 



(Less Favorable) 



(liore Favorable) 



Unfortunately, the wording ^>f the item itself is rather long 
and perhaps not altogether lucid. A student who did no*, read the 
item carefully might have responded positively or negatively on 
the basis of his/her feeling abqut the course as a whole rather 
.than on the basis of the uncontrollable factors.' If this hap- 
pened with moderate frequency, the results would be correlations 
of the magnitude and direction obtained (p . 2) . 

. Given the potential ln5>ortance of the finding described above, we 
designed a follow-up study to more clearly explicate 'the relationship of 
the ratings of factors outside the instructor's control with the ratings 
of evaluative items, /j 

I METHOD 
The Inatruments - * 

Fgll Quarter of 1974, all teaching faculty in the Department of 
Anthropology .were asked to administer the' Student Ratings Forms of the 
Instructional Assessment System (Gillmore, Note 2) . The Instructional Assess- 
ment System provides five different forms for instructor use; however, all 
forms share four general, evaluative items. These four items and their 
response positions are found in Table 2. < 

In addition to the regular form, seven additional items, designed to 
clarify the puzzling relationship noted above, were administered to students. 
These items with their response positions are found in Table 3. Items 23 
through 26 are specific demographic features of a course which' are con- 
sidered to be out of the instructor's control. Item 27 is simply a rewording 
of the third 'item of the previous study, but with the ambiguity removed by 
only asking if there was a tendency to rate the course lower because of. 
circumstances beyond the instructqij's control. The wording of item 28 repre- 
sents an attempt to neutralize factors outside the instructor's control by - 
asking 'students how they would expect- to. -rate a different, equally qualified 



Table 2 

^ Global, Evaluative Iteae of the Inatructlonal Ass^sment System 

♦ 

1« The course as a whole was: ' E' VG 6 F 

2. 1%e course content was:. B VG G F 
Tha Jbistructor's c^^ E VG G P 

'4* tba Instructor's effectiveness In teaching 

the subject patter was: f*. E- VG G P 



E « Excellent 
VG » Very Good 

G « Good 

F • Fair ^ 

P » Poor 
VP « Very Poor 



. Table 3 
Additional iteas, J«ll Quarter 1974 



23. The sire of thllj class iaT: * E VG G f P yp . 

.24. Ihe_tlme of day at which the class meets is: E V6 G P P VP ' 

25. The location of the class is: E VG ,G F P VP 

26. JUie physical qualities of the classroom are; E VG G F P VP 

' * ■ " > ■ 

27. Do you feel that there were circumstances beyond the instructor 'i 
control, such as the four above, which caused you to evaluate this 
course less favorably than you might have otherwise? 

0 « Definitely not • ' '. ♦ 

1 » Probably not . ^ 

2 « Possibly ' , , 

3 - Probably - . • , , 

4 » Definitely ' . " * 

28* If everything else about the course were the same, how'would you 
«tgect to ifate a dif ferenti equaUy-qualiflid Instructor? 

0, « Much more favorably than I rated the present Instructor 

1 » Somewhat higher v 

2 « Probably aboutj the same . ^ . 

3 * Somewhat lower ^' 

4 » I would rate^Jy present instructor higher • 



' i . c - • , • 

29e Do you think that this course is one whfch Is fairly easy to teach? 



0 « Yes j 

1 » About average 

2 » NTo ' • I ^ 

3 •* I really have no idea 



B • Excellent 
VG «■ Very Good 

& » Good 

P « Pair 

P « Poor, 
VP « Very Poor 



Instructor teaching the swe course. Finally, item 29 I|tten5>t8 a global 
asseaasent of factors outside the instructor's* control by uking about how 
easy the course is to teach. « 

Also included in the analysis to b^ presented subsequently was an item 
found at the top of all student ratings forms: When enrolling, was this a 
course you wanted to take: Yes, Neutral, No. Class size, as indicated 
by the nunier of forms conpleted for a class, was also recorded for subse- 
quent analysis. ' ' ' 
Data Source 

The instrument desc|;ibed above was administered in twenty-eight*' 
Anthropology classes. The smallest class size was three students, and the 
largest was 147, with a mean class size of 51 students and a median class 
size o'f 11.5. 

' ' . RESULTS 

All itepa from the standard Student Ratings ^orms were correlated 
with, the seven additional items. The seven additional items were corre- 
la ted among themselves. ^Also correlated with the above was the item "Was 
this a cfourse you wanted to take?" and class size,. It is in^ortant to 
note that these* coinrelations were calculated across the 28 class means, 
not individual students. These' correlations are presented* in Table 4. 
Only correlations involving fhe 'four global items of the standard rating 
form^ are {^resented, since correlations involving the remaining items are 
completely consistent with these fout and add little information. , 
Factors Outside the Ingtructor^s Cofattol 

The four items tapping students* attitude toward specific demographic 
features of the classes correlated moderately among themselves (.29 to .58). 



1 

2 

3 

4 

23 

24 

25 

26 

27 

28 

29. 



23 

, i 
-* 06 

-.09 

-02 

. 06 



* . V ■ ' Table , * ' 

Colf^aliatlons amoag Student Ratings Items* , 



24. 

♦ 

•1 

24 

05 . 

11 
20 

44*- 



% J 



/ 



-06- 
' -09 ( 
'10 



26 

17 
00 
14" 



5Sr^ 29, 

♦ 

47* ' 48* 



27 . 


' 28 


29 


''Wanted 
to take 
class 


Cl«88 

Size- « 




47* 


21 


14 / 
> 


-03 


03 


38* 


24 


15 


-01 


-03 


. 55* 

\ - 


15 


11 


06 




50* 

-15 • ' 


19 


19 


08 


-65* ' 


-04 


• 04 


-63* . 


-51* 




^16 


04 


-30 


^74* 


17 


26 


34 


-13 


-34 


•29 


-67 


37 


13 




-19 * 


-15 


-11 


28 






41* 


18* 
24 


20 

17 / 






f 







*DeclnaI points have been eliminated for ease of reading. 
*P .05 / , . 
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The, Mtlng.of the favorabUness of the class site correlated with ai^tual 
class size (^.63), In'dicatlng that students: teJded to like smsll^r classes.* 



The repalnlng correlations with class sLe we?i ;nuch* smaUer. Thus, within 
this simple of classes, the ratlirg^ of kbt foui demographic features were 
'•related, however, this relationship cankot be attributed 'to the size of 

the class alone ~ ' . . ' • 

i , ■ ■ 

All four of the deaographic items 'cofrelafced with the more global item 
27, dealing with an overall assessment of the Ufegt.of factprs outside the 
. instructor!8^cpntro\'(.-.63, -.-SI,',-. 74, an^ -.34). ' This Jives evidence that 
-much of what influen.ced students in the ratlnJ of^ item 27 was captured in 
items 23, . 24, and 25,."and to a less^ 
curiously^ was the location of the class 

^latlon between Factors Outside the Instnictkr's Control and Overall 

" \ ' ^ — : ^ 

' Evaluatloo 

None'of the .first five special Items cor relate^ significantly with the '* 



extent- 26. The highest correlation. 



^our global evaluatlonal items. Furthermore, 



with item 28, which was ai^ explicit atttempt t^o neutralize the effect of 



none corr^lajted significantly 



em 28 did, bowever, correlate 
tt-dld serve to dome exteht • 



\ factors outside. the Instructoir's control. It 
with Items one through, four, Which Indil^ates 
as a .general evaluative Item. 

Item 29 asked the student's perceptl^ o 
teach. This item apparently failed as an 

' outside the instructtj^'s^ontrol,,; since it ^^d to correlate signifi- 
cantly with any of the items dealing directlyWi^h these factors. Nor did 

this item correlate significantly with any of ke four global evaluation 

' \ > i 

items i thus; it did not seem to tap the student'^ dverall evaluation of the ' 



I of 
\ndi; 



how easy the course was to 
ect assessment of iactors 
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course. * The only correlation of moderate size was^with Item 28 (.Al), ' 
showing a tendency for a course rated as easier to teach to also be one in 

" • • • / . - . ^ : . - 

which another instructor would be -rated higher as well. ' - v 

Fihally,\the item •'When enrolling, was this a course you wanted to 
take?" did riot correlate significantly wit» any of the general evaluation 
items, or any of the factots outside the Instructor's control* Thus," this 
sample gives no evidence that ^recourse attitudes, as measured by this ^ 
item, are influential ijx postcourse ratings. " 

— ^ Discussion ^ ' " 

This study set out t(^g^n a more accurate appraisal of the/ effect of 
factors outside Jhe tast^^ control on ratings by students' of his/her 

Instruction. The xresults of the study seem to*strongly indicate that these 

; ' ' / ' 

factorst do not have an influence. As direct evidence. Items 23 through 27 

did not correlate significantly with any bi^ the global' evaluative items. ' 

T6e lack of relationship between" item 27 and the gl^>^ evaliiative items Is 

especallly important, since it was a similar item which showed significant 

correlations in the p^revlous study^It .looks as though the correlations 

foudd in that study were caused by the particular^ item wording rather than 

by any actual relationship. ^^'^ • 

" » 

As further evidence, it is instructive to Ipok closely at the resxilts' 

' • ^-^ . * * - 

of Item 28. In terms of assessing the influ,ence''of factors outside the 

instructor's control rfel'ative to the influence of his/her actual^ perform- 

ance, there seems^to be two distinct outcomes for this item. Insofar as * 

factors outsidfe the instructor's control have an influence, the^ rating of a 

different, eqt^lly-qualified instructor would not seem to^^ange, since he/she 



iMuId-be paddled with these eaoe fectore. fnaofer «s the lnatructor'A • ' 
I^cforiMnbr is tofluential, the cojirse would .be rated as Ijetter^uught by - 
, soaarn^ else in the case of a po^or instructor; and jbife pctorly taught by 
•OMone else ift «he case of agobd instructof. ' 

For t& tirst possibility,. item 28 would correlate. with itei^^ 

. not with iteaa one tlirough four (especially three, and four which are global * 

"^j— ■■ ' ■ ' ^ ' '• ■ . - ^ ■• - • . 

■y^ instructor items) . For, the second pos^ibUity^ tl^c opposite "result is - 

' • • , ■ ■ * • " , • • ■' • ■ • 

c predictable* The- data from this study coine down in favo> of the latter. ^ • 

• The ^rrelation .between items 28 and^2? is' low and non-significant (-.19). 
The correlations , between item j28 and iteiis 3 and 4ar6much higher and sig- 
nific^t (.55 'and .50 respectively). s . ' ' - 

• Although the data appeat to give aT;*denire that factors' outside the ' 
^instructor's control are not a. sig^ficant determiner of global student 
f ratings,- some 'basi<i limitations in the method of the. study should be men- . ^ 
tioned. First, the study was \ done in only one academic department. Thus.i' " 

these results cannot justifiably be generalized to all disciplines without 
■ ' • . ■ • ■ , •. ■ ' ' 

teplication to a more representative set of departments. On the other hand, 

there is no clear reason to expect Anthrbpolo^r classes, to differ in any 

isflpbrtant way from classes at least withto other social science discipline 

v^rees'. • . 

^ Secondly, the nunibfer of classes represented in the study is relatively 
small (28) and many of these classes were quite small, A case cat easily 
be made for rei^licatlng this study in a larger 8an?>le, as well as one mor6 
representative of j*e entire University. 

Thiied, one should not Ipse sight of the. fact tlfat this study is based 
on the student's perception of the-tofluence of factors outside the 
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instructor's ' control y not the actual factors themselves* The only variable 
In this study -otf actual, as opposed to rated, outside factors is class ^ize/ 
.vhich <ild not correlate •^ignificantly with the- evaluative, itfeas, ^Other 

factors might, howevfer* . * ' 

' ' • • *» ' * * ^. . - - 

Finally,' the basic design of this study was correlational, which ma^es 

cause and elffect statements tenuous at best. Future studies could be 

designed which actually raanipJiate factors of interest, such as the '<ilas8- 

ropm in which the classes are held. * . 3 - ' ^ 
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